Response to Barnett et al. Small N designs for rehabilitation research. J Rehabil Res Dev. 2012;49(1):175-86.

I am writing to congratulate Bar-nett et al. for their article on small N designs, as well as the Journal of Rehabilitative Research and Development for publishing it. Their article fosters an opportunity to formalize the role of pilot studies in rehabilitation research. I believe that there will always be room to improve the efficiency with which novel rehabilitation therapies are developed and advocated. Adequate or perhaps even overwhelming evidence for efficacy is necessary to promote a new therapy if it costs time or money, proposes to replace existing treatments, or causes undesirable side effects. Efficacy is traditionally demonstrated with the randomized controlled trial (RCT), but difficulties in recruitment and retention, problems shared with the pharmaceutical industry [1], often lead to underpowered studies. Furthermore , the background of many rehabilitation interventions do not approach the theoretical foundations of drug studies in which causal mechanisms understood at the molecular level allow one to show drug efficacy in spite of weak design and analysis [2]. These challenges, and others identified by Barnett et al. [3], result in inefficient rehabilitation therapeutic development and promotion in the Veterans Health Administration. Barnett et al. recognize that, in general, small N or single-case designs alone may not provide sufficient evidence to promote a new therapy. However, small N designs in the context of pilot studies may greatly enhance the efficiency with which new therapies are discovered before testing in a conventional, large-scale, randomized trial. One might think of a pilot study as an attempt to identify the one therapy among a set of competitors that is most likely to deliver positive results in a large-scale randomized trial. Consider the following design: subjects recruited into the pilot study are randomly assigned to competitors and consistently measured according to the single-case protocol described in Barnett et al. with clearly defined outcome measures. After an arbitrary period of recruitment, the collection of single-case recruitments is analyzed using meta-analysis. Based on the analysis, competitors are ranked in terms of average outcome so that each is assigned a probability of being the most effective among therapies considered. The pilot study continues until the investigator is sufficiently convinced that the best competitor has been identified and, perhaps most importantly, believes that the results of the study will convince reviewers of the value of a large-scale random-ized trial. I would recommend a Bayesian hierarchical modeling approach that allows one to use simple , …